[Docs] Add user guide for configuring authentication for Ray clusters #49080

andrewsykim · 2024-12-04T20:36:31Z

Why are these changes needed?

Add a user guide for configuring authenticadtion and access control for Ray clusters using KubeRay and Kubernetes RBAC.

Related issue number

Based on REP: https://github.com/ray-project/enhancements/blob/main/reps/2024-05-21-kuberay-authentication.md

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Moonquakes · 2024-12-05T01:21:34Z

Hi @andrewsykim , I found that the protection in the document is mainly for submitting ray jobs through port 8265. Is there a corresponding protection mechanism and document for interactive access to Ray Cluster through port 10001? (It seems that the ray.init interface needs to be modified, but I am not sure if it is currently supported)

andrewsykim · 2024-12-05T02:17:29Z

@Moonquakes that's correct, for now we are starting with authentication for Ray dashboard only, in the future we will consider client port. @kevin85421 suggested we actually stay away from recommending interactive client but I can't recall the reasons

kevin85421

Nice!

kevin85421 · 2024-12-09T17:25:18Z

doc/source/cluster/kubernetes/user-guides.md

@@ -27,6 +27,7 @@ user-guides/tls
 user-guides/k8s-autoscaler
 user-guides/static-ray-cluster-without-kuberay
 user-guides/kubectl-plugin
+user-guides/kuberay-auth


Also, add:

* {ref}`kuberay-auth`

below. Currently, the page only appears in the sidebar.

kevin85421 · 2024-12-09T17:28:05Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+## Prerequisites
+
+*   A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).


Suggested change

* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).

* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).

kevin85421 · 2024-12-09T17:28:15Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+## Prerequisites
+
+*   A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).
+*   `kubectl` installed and configured to interact with your cluster.


Suggested change

* `kubectl` installed and configured to interact with your cluster.

* `kubectl` installed and configured to interact with your cluster.

kevin85421 · 2024-12-09T17:28:23Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+*   A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).
+*   `kubectl` installed and configured to interact with your cluster.
+*   `gcloud` CLI installed and configured (if using GKE).


Suggested change

* `gcloud` CLI installed and configured (if using GKE).

* `gcloud` CLI installed and configured (if using GKE).

kevin85421 · 2024-12-09T17:28:30Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+*   A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).
+*   `kubectl` installed and configured to interact with your cluster.
+*   `gcloud` CLI installed and configured (if using GKE).
+*   Helm installed.


Suggested change

* Helm installed.

* [Helm](https://helm.sh/) installed.

kevin85421 · 2024-12-09T17:28:38Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+*   `kubectl` installed and configured to interact with your cluster.
+*   `gcloud` CLI installed and configured (if using GKE).
+*   Helm installed.
+*   Ray installed locally.


Suggested change

* Ray installed locally.

* Ray installed locally.

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

kevin85421 · 2024-12-09T19:08:08Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+Get a token for the `ray-user` service account and store it in the `RAY_JOB_HEADERS` environment variable:
+
+```bash
+export RAY_JOB_HEADERS="{\"Authorization\": \"Bearer $(kubectl create token ray-user)\"}"


Can you add a note that kubectl create token is only supported in kubectl versions >= 1.24 (ref) or update the "Prerequisites" section?

1.24 has been EOL for a while now, it's probably safe to assume most users are using a newer version at this point?

I followed this documentation and realized that I am using version 1.23.X on the client side. On the server side, I think most people use newer Kubernetes versions, but on the client side, users might not upgrade as frequently.

good point, I'll add a note about this

kevin85421 · 2024-12-09T19:10:23Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+# Configure Ray Clusters with Authentication and Access Control using KubeRay
+
+This guide demonstrates how to secure Ray clusters deployed with KubeRay by enabling authentication and access control using Kubernetes Role-Based Access Control (RBAC).


Can you add the current limitations to the documentation? For example, it currently doesn't support RayJob and RayService. How can I use the browser to view the dashboard? (follow up)

ack, I can also add some instructions for accessing the dashboard via browser

added instructions for using browser to access Ray dashboard

kevin85421 · 2024-12-09T19:17:45Z

@Moonquakes, yeah, the Ray maintainers currently recommend that users avoid using the Ray client. It has not been actively maintained for years due to some fundamental stability issues. We have been discussing Ray Client V2, but it is still in the planning stage for now.

Signed-off-by: Andrew Sy Kim <[email protected]>

kevin85421 · 2024-12-09T21:10:51Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+This guide demonstrates how to secure Ray clusters deployed with KubeRay by enabling authentication and access control using Kubernetes Role-Based Access Control (RBAC). 
+
+> **Note:** This guide is only supported for the RayCluster custom resource.


How long does the token last before it expires? I followed the documentation before lunch, but after lunch, the token stopped working, and I had to create a new one. Not necessary to address this in this PR.

I believe tokens last 1hour by default, you can specify the duration though:

kubectl create token ray-user --duration=24h

The server may return an expiration shorter than what the client requests though

Would you mind adding some comments regarding the expiration? Thanks!

Updated the example to include --duration so users know the flag exists

kevin85421 · 2024-12-09T21:11:26Z

I will ping our doc team to review this PR.

Signed-off-by: Andrew Sy Kim <[email protected]>

angelinalg · 2024-12-12T01:42:47Z

Thank you for contributing to Ray docs. Just some style nits to be more consistent with our style guide. Thank you.

andrewsykim · 2024-12-12T03:36:42Z

@angelinalg I didn't see any comments for style nits, did you forget to submit the review?

angelinalg · 2024-12-12T01:36:08Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

@@ -0,0 +1,207 @@
+(kuberay-auth)=
+
+# Configure Ray Clusters with Authentication and Access Control using KubeRay


Suggested change

# Configure Ray Clusters with Authentication and Access Control using KubeRay

# Configure Ray clusters with authentication and access control using KubeRay

Trying to standardize on sentence case for titles.

angelinalg · 2024-12-12T01:36:48Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+## Prerequisites
+
+* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).


Suggested change

* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).

* A Kubernetes cluster. This guide uses GKE, but the concepts apply to other Kubernetes distributions.

Avoiding excessive use of parentheses.

angelinalg · 2024-12-12T01:37:13Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).
+* `kubectl` installed and configured to interact with your cluster.
+* `gcloud` CLI installed and configured (if using GKE).


Suggested change

* `gcloud` CLI installed and configured (if using GKE).

* `gcloud` CLI installed and configured, if using GKE.

angelinalg · 2024-12-12T01:37:31Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+* [Helm](https://helm.sh/) installed.
+* Ray installed locally.
+
+## Create a GKE Cluster (or use an existing cluster)


Suggested change

## Create a GKE Cluster (or use an existing cluster)

## Create or use an existing GKE Cluster

angelinalg · 2024-12-12T01:37:52Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+## Create a GKE Cluster (or use an existing cluster)
+
+If you don't have a Kubernetes cluster, create one using the following command (or adapt it for your cloud provider):


Suggested change

If you don't have a Kubernetes cluster, create one using the following command (or adapt it for your cloud provider):

If you don't have a Kubernetes cluster, create one using the following command, or adapt it for your cloud provider:

angelinalg · 2024-12-12T01:40:24Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+The output should be `yes`.
+
+## Submit a Ray Job with Authentication


Suggested change

## Submit a Ray Job with Authentication

## Submit a Ray job with authentication

angelinalg · 2024-12-12T01:40:57Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+ray job submit --address http://localhost:8265  -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
+```
+
+The job should now succeed, and you'll see output similar to this:


Suggested change

The job should now succeed, and you'll see output similar to this:

The job should now succeed, and you should see output similar to this:

angelinalg · 2024-12-12T01:41:23Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+
+## Verify Access Using Cloud IAM (Optional)
+
+Most cloud providers allow you to authenticate to your Kubernetes cluster as your cloud IAM user. This is a convenient way to interact with the cluster without managing separate Kubernetes credentials.


Suggested change

Most cloud providers allow you to authenticate to your Kubernetes cluster as your cloud IAM user. This is a convenient way to interact with the cluster without managing separate Kubernetes credentials.

Most cloud providers allow you to authenticate to the Kubernetes cluster as your cloud IAM user. This method is a convenient way to interact with the cluster without managing separate Kubernetes credentials.

angelinalg · 2024-12-12T01:41:34Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+------------------------------------------
+```
+
+## Verify Access Using Cloud IAM (Optional)


Suggested change

## Verify Access Using Cloud IAM (Optional)

## Verify access using cloud IAM (Optional)

angelinalg · 2024-12-12T01:42:03Z

doc/source/cluster/kubernetes/user-guides/kuberay-auth.md

+ray job submit --address http://localhost:8265  -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
+```
+
+The job should succeed if your cloud user has the necessary Kubernetes RBAC permissions (you may need to configure additional RBAC rules for your cloud user).


Suggested change

The job should succeed if your cloud user has the necessary Kubernetes RBAC permissions (you may need to configure additional RBAC rules for your cloud user).

The job should succeed if your cloud user has the necessary Kubernetes RBAC permissions. You may need to configure additional RBAC rules for your cloud user.

angelinalg · 2024-12-12T17:16:08Z

So sorry for the oversight, @andrewsykim!
And BTW, hello! Hope you're doing well.

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim · 2024-12-12T21:29:01Z

Thanks @angelinalg! I'm doing great, hope you're doing well too! :)

Moonquakes · 2024-12-13T01:08:56Z

@Moonquakes, yeah, the Ray maintainers currently recommend that users avoid using the Ray client. It has not been actively maintained for years due to some fundamental stability issues. We have been discussing Ray Client V2, but it is still in the planning stage for now.

Thanks for your answer! However, Ray Client will be very helpful in the scenario of using Jupyter, which can greatly improve development efficiency. I look forward to the release of Ray Client v2!

kevin85421 · 2024-12-13T05:22:16Z

Ray Client will be very helpful in the scenario of using Jupyter, which can greatly improve development efficiency.

Agree. A good Ray client provides better UX comparing with ray job submit.

…ray-project#49080) Signed-off-by: Andrew Sy Kim <[email protected]> Signed-off-by: ujjawal-khare <[email protected]>

andrewsykim requested review from maxpumperla, pcmoritz, kevin85421 and a team as code owners December 4, 2024 20:36

kevin85421 self-assigned this Dec 4, 2024

andrewsykim force-pushed the kuberay-auth branch from 9b12bff to 4635d69 Compare December 5, 2024 02:25

kevin85421 reviewed Dec 9, 2024

View reviewed changes

[Docs] Add user guide for configuring authentication for Ray clusters

da03f2c

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the kuberay-auth branch 3 times, most recently from 8a2029f to 2b643b7 Compare December 9, 2024 20:58

kevin85421 approved these changes Dec 9, 2024

View reviewed changes

add pre-requisite and improve formatting

04201dd

Signed-off-by: Andrew Sy Kim <[email protected]>

andrewsykim force-pushed the kuberay-auth branch from 2b643b7 to 04201dd Compare December 10, 2024 20:10

andrewsykim mentioned this pull request Dec 11, 2024

[Umbrella] KubeRay Authentication Support ray-project/kuberay#2581

Open

7 tasks

angelinalg reviewed Dec 12, 2024

View reviewed changes

address PR review comments

154f5fe

Signed-off-by: Andrew Sy Kim <[email protected]>

kevin85421 added the go add ONLY when ready to merge, run all tests label Dec 13, 2024

kevin85421 approved these changes Dec 13, 2024

View reviewed changes

jjyao merged commit 7626673 into ray-project:master Dec 13, 2024
6 checks passed


		## Prerequisites

		* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).

	* `kubectl` installed and configured to interact with your cluster.
	* `kubectl` installed and configured to interact with your cluster.

	* `gcloud` CLI installed and configured (if using GKE).
	* `gcloud` CLI installed and configured (if using GKE).


		# Configure Ray Clusters with Authentication and Access Control using KubeRay

		This guide demonstrates how to secure Ray clusters deployed with KubeRay by enabling authentication and access control using Kubernetes Role-Based Access Control (RBAC).


		This guide demonstrates how to secure Ray clusters deployed with KubeRay by enabling authentication and access control using Kubernetes Role-Based Access Control (RBAC).

		> Note: This guide is only supported for the RayCluster custom resource.

		@@ -0,0 +1,207 @@
		(kuberay-auth)=

		# Configure Ray Clusters with Authentication and Access Control using KubeRay

	* A Kubernetes cluster (this guide uses GKE, but the concepts apply to other Kubernetes distributions).
	* A Kubernetes cluster. This guide uses GKE, but the concepts apply to other Kubernetes distributions.

	* `gcloud` CLI installed and configured (if using GKE).
	* `gcloud` CLI installed and configured, if using GKE.

	## Create a GKE Cluster (or use an existing cluster)
	## Create or use an existing GKE Cluster


		## Create a GKE Cluster (or use an existing cluster)

		If you don't have a Kubernetes cluster, create one using the following command (or adapt it for your cloud provider):


		The output should be `yes`.

		## Submit a Ray Job with Authentication

	The job should now succeed, and you'll see output similar to this:
	The job should now succeed, and you should see output similar to this:


		## Verify Access Using Cloud IAM (Optional)

		Most cloud providers allow you to authenticate to your Kubernetes cluster as your cloud IAM user. This is a convenient way to interact with the cluster without managing separate Kubernetes credentials.

	The job should succeed if your cloud user has the necessary Kubernetes RBAC permissions (you may need to configure additional RBAC rules for your cloud user).
	The job should succeed if your cloud user has the necessary Kubernetes RBAC permissions. You may need to configure additional RBAC rules for your cloud user.

[Docs] Add user guide for configuring authentication for Ray clusters #49080

[Docs] Add user guide for configuring authentication for Ray clusters #49080

Conversation

andrewsykim commented Dec 4, 2024

Why are these changes needed?

Related issue number

Checks

Moonquakes commented Dec 5, 2024

andrewsykim commented Dec 5, 2024

kevin85421 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewsykim Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 commented Dec 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevin85421 commented Dec 9, 2024

angelinalg commented Dec 12, 2024

andrewsykim commented Dec 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angelinalg commented Dec 12, 2024 • edited Loading

andrewsykim commented Dec 12, 2024 • edited Loading

Moonquakes commented Dec 13, 2024

kevin85421 commented Dec 13, 2024

andrewsykim Dec 9, 2024 •

edited

Loading

andrewsykim Dec 9, 2024 •

edited

Loading

angelinalg commented Dec 12, 2024 •

edited

Loading

andrewsykim commented Dec 12, 2024 •

edited

Loading